Method for the 3d reconstruction of a scene

ABSTRACT

The present invention concerns a method for the 3D reconstruction of a scene comprising the matching ( 610 ) of a first event from among the first asynchronous successive events of a first sensor with a second event from among the second asynchronous successive events of a second sensor depending on a minimisation ( 609 ) of a cost function (E). The cost function comprises at least one component from: —a luminance component (E i ) that depends at least on a first luminance signal (I u ) convoluted with a convolution core (gσ(t)), the luminance of said pixel depending on a difference between maximums (t e−,u ,t e+,u ) of said first signal; and a second luminance signal (I v ) convoluted with said convolution core, the luminance of said pixel depending on a difference between maximums (t e−,v ,t e+,v ) of said second signal; —a movement component (E M ) depending on at least time values relative to the occurrence of events located at a distance from a pixel of the first sensor and time values relative to the occurrence of events located at a distance from a pixel of the second sensor.

The present invention relates to the area of the 3D reconstruction of a scene, in particular when it is captured using asynchronous sensors.

Contrary to standard cameras which record successive images at regular sampling moments, the biological retinas only transmit a little repetitive information about the scene to view, and this, asynchronously.

Event-based asynchronous vision sensors deliver compressed digital data in the form of events.

A presentation of such sensors can be viewed in “Activity-Driven, Event-Based Vision Sensors”, T. Delbrück, et al., Proceedings of 2010 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 2426-2429. Event-based vision sensors have the advantage of increasing repetition, reducing latency time and increasing the range of time dynamics and grey levels compared with standard cameras.

The output of such a vision sensor can consist, for each pixel address, of a sequence of asynchronous events, representative of changes in the reflectance of the scene at the time they are produced.

Each pixel of the sensor is independent, and detects changes in light intensity higher than a threshold, from the transmission of the last event (for example, a contrast of 15% on the intensity logarithm). When the change in intensity exceeds the threshold set, an ON or OFF event is generated by the pixel according to whether the intensity increases or decreases (DVS sensors). Certain asynchronous sensors connect detected events to absolute measurements of light intensity (ATIS sensors).

The sensor, not being sampled on a clock like a standard camera, it can chart the sequencing of events with a very high time precision (for example, around 1 μs). If such a sensor is used to reconstruct a sequence of images, a rate of images of several kilohertz can be achieved, compared with a few dozen hertz for standard cameras.

Moreover, in the framework of the 3D reconstruction of a scene, for each one of the pixels of the sensors, a calculation of the position in space is made. In order to achieve this, there are many methods using several cameras or other standard sensors. Consequently, these methods achieve determinations using standard 2D images wherein the pixels have at least one value (that is, they are defined).

For asynchronous sensors, such as previously defined, such methods cannot be applied by definition, as no “standard” 2D image is not available exiting the sensors: in order to use these methods, it would be necessary to artificially “reconstruct” 2D images from asynchronous information from the sensors. However, this reconstruction can be heavy and handling complete images can require consequent processing means. In addition, this reconstruction discretises time information and thus, the time dependency of the visual information is practically ignored.

Consequently, there is a need to develop 3D scene reconstruction methods, methods suitable for asynchronous sensors.

The present invention aims to improve the situation.

To this end, the present invention proposes a method, especially suited to asynchronous sensors to reconstruct scenes observed in 3D.

The present invention therefore aims for a method for the 3D reconstruction of a scene, the method comprising:

-   -   reception of a first piece of asynchronous information from a         first sensor that has a first pixel matrix positioned opposite         the scene, the first piece of asynchronous information         comprising, for each pixel of the first matrix, first successive         events coming from said pixel;     -   reception of a second piece of asynchronous information from a         second sensor that has a second pixel matrix positioned opposite         the scene, the second piece of asynchronous information         comprising, for each pixel of the second matrix, second         successive events coming from said pixel, the second sensor         being separate from the first sensor;     -   matching of a first event from amongst the first successive         events with a second event from amongst the second successive         events according to a minimisation of a cost function;

wherein the cost function comprises at least one component from amongst:

-   -   a luminance component, said luminance component depending on at         least:         -   one first luminance signal coming from a pixel from the             first convoluted sensor with a convolution core, the             luminance of said pixel depending on a difference between             the maximums of said first signal; and         -   one second luminance signal coming from a pixel from the             second convoluted sensor with said convolution core, the             luminance of said pixel depending on a difference between             the maximums of said second signal;     -   a movement component, said movement component depending on at         least:         -   time values relating to the occurrence of events spatially             located at a predetermined distance from a pixel of the             first sensor;         -   time values relating to the occurrence of events spatially             located at a predetermined distance from a pixel of the             second sensor.

Thus, it is not necessary, for the reconstruction of 3D scenes captured using several DVS or ATIS asynchronous sensors, to recreate standard 2D images for using methods of the prior art applicable to these images.

Consequently, the precision of such a 3D reconstruction is very precise/large, the asynchronous time information being a lot more precisely sampled.

Moreover, the cost function can additionally comprise:

-   -   a time component, said time component depending on a difference         between:         -   a time value relating to an event of the first sensor;         -   a time value relating to an event of the second sensor.

It is therefore possible to avoid events that are too separate time-wise being connected.

In a specific embodiment, the cost function can additionally comprise:

-   -   a geometric component, said geometric component depending on:         -   a spatial distance of a pixel of the second sensor at an             epipolar straight line or at an intersection of epipolar             straight lines defined by at least one pixel from the first             sensor.

It is therefore possible to avoid events not corresponding to the same point X(t) of the scene being connected.

Advantageously, the luminance signal of the pixel of the first sensor and of the pixel of the second sensor comprising a maximum, coding an occurrence time of a luminance variation, the convolution core can be a predetermined Gaussian variance.

In a specific embodiment, said luminance component can additionally depend on:

-   -   luminance signals of pixels of the first sensor, spatially         located at a predetermined distance from the first pixel of the         first sensor, convoluted with the convolution core; and     -   luminance signals of pixels of the second sensor, spatially         located at a predetermined distance from the second pixel of the         second sensor, convoluted with the convolution core.

Consequently, considering events close to pixels to be connected enables to check if the whole corresponds, and check that the fact of obtaining a local correspondence for two pixels is not a simple artefact or a simple singularity.

In addition, said movement component can additionally depend on:

-   -   an average value of the time values relating to the occurrence         of pixel events of the first sensor, spatially located at a         predetermined distance from the pixel of the first sensor;     -   an average value of the time values relating to the occurrence         of pixel events of the second sensor, spatially located at a         predetermined distance from the pixel of the second sensor.

In a specific embodiment, said movement component can depend on, for a given time:

-   -   for each current time value relating to the occurrence of events         spatially located at a predetermined distance from a pixel of         the first sensor, a function value, decreasing from a distance         of said given time to said current time value;     -   for each current time value relating to the occurrence of events         spatially located at a predetermined distance from a pixel of         the second sensor, a function value, decreasing from a distance         of said given time to said current time value.

In an alternative embodiment, said movement component can depend on:

-   -   a first convolution of a decreasing function with a signal         comprising a Dirac for each time value relating to the         occurrence of events spatially located at a predetermined         distance from a pixel of the first sensor;     -   a second convolution of a decreasing function with a signal         comprising a Dirac for each time value relating to the         occurrence of events spatially located at a predetermined         distance from a pixel of the second sensor.

The present invention also aims for a device for the 3D reconstruction of a scene, the method comprising:

-   -   an interface for receiving a first piece of asynchronous         information from a first sensor that has a first pixel matrix         positioned opposite the scene, the first piece of asynchronous         information comprising, for each pixel of the first matrix,         first successive events coming from said pixel;     -   an interface for receiving a second piece of asynchronous         information from a second sensor that has a second pixel matrix         positioned opposite the scene, the second piece of asynchronous         information comprising, for each pixel of the second matrix,         second successive events coming from said pixel, the second         sensor being separate from the first sensor;     -   a processor suitable for matching a first event from amongst the         first successive events with a second event from amongst the         second successive events depending on a minimisation of a cost         function;

wherein the cost function comprises at least one component from amongst:

-   -   a luminance component, said luminance component depending on at         least:         -   a first luminance signal coming from a pixel of the first             sensor convoluted with a convolution core, the luminance of             said pixel depending on a difference between maximums of             said first signal; and         -   a second luminance signal coming from a pixel of the second             sensor convoluted with said convolution core, the luminance             of said pixel depending on a different between the maximums             of said second signal;     -   a movement component, said movement component depending on at         least:         -   time values relating to the occurrence of events spatially             located at a predetermined distance from a pixel of the             first sensor;         -   time values relating to the occurrence of events spatially             located at a predetermined distance from a pixel of the             second sensor.

A computer program, implementing all or part of the method described above, installed on pre-existing equipment, is advantageous in itself.

Thus, the present invention also aims for a computer program comprising instructions for the implementation of the method previously described, when this program is executed by a processor.

This program can use any programming language (for example, object language or other), and be in the form of an interpretable source code, a partially compiled code or a totally compiled code.

FIG. 6 described in detail below, can form the general algorithm flowchart of such a computer program.

Other characteristics and advantages of the invention will again appear upon reading the description that will follow. This is purely illustrative and must be read while looking at the appended drawings whereon:

FIG. 1 is an overview drawing of an ATIS asynchronous light sensor;

FIG. 2 is a diagram showing the events generated by an asynchronous sensor positioned opposite a scene comprising a rotating star;

FIG. 3 is an example of the calculation of a luminance component for two points of two separate sensors;

FIGS. 4a and 4b are representative examples of an activity signal of a given pixel;

FIG. 4c μs a representation of movement cards generated using separate asynchronous sensors;

FIGS. 5a and 5b are representations of examples of calculations of geometric components in an embodiment of the invention;

FIG. 6 illustrates a flowchart presenting an embodiment according to the invention;

FIG. 7 illustrates a device for implementing an embodiment according to the invention.

FIG. 1 illustrates the ATIS principle.

A pixel 101 of the matrix constituting the sensor comprises two photosensitive elements 102 a, 102 b, such as photodiodes, respectively connected to electronic detection circuits 103 a, 103 b.

The sensor 102 a and its circuit 103 a produce a pulse P₀ when the light intensity received by the photodiode 102 a varies from a predefined quantity.

The pulse P₀ marking this change in intensity triggers the electronic circuit 103 b connected to the other photodiode 102 b. This circuit 103 b then generates a first pulse P₁ then a second pulse P₂ as soon as a given quantity of light (number of photons) is received by the photodiode 102 b.

The time difference δt between the pulses P₁ and P₂ is inversely proportional to the light intensity received by the pixel 101 just after the appearance of the pulse P₀.

The asynchronous information coming from the ATIS comprises two combined pulse trains for each pixel (104): the first pulse train P₀ indicates the moments when the light intensity has changed beyond the detection threshold, while the second train is composed of pulses P₁ and P₂ of which the time difference δt indicates corresponding light intensities or grey levels.

An event e(p, t) coming from a pixel 101 of position p in the matrix in the ATIS thus comprises two types of information: a time-related piece of information given by the position of the pulse P₀, giving the moment t of the event, and a piece of grey level information given by the time difference δt between the pulses P₁ and P₂.

Events coming from pixels in a three-dimensional space/time representation such as that presented in FIG. 2 can then be placed. In this figure, each point p identifies an event e(p, t) asynchronously generating a moment t at the level of a pixel p of the sensor, of position=(_(y) ^(z)), by the movement of a star rotating at a constant angular speed as shown in a diagram in box A. The major part of these points is distributed close to a surface of a general helicoidal form. In addition, the figure shows a certain number of events at a distance from the helicoidal surface, which are measured without corresponding to the actual movement of the star. These events are from the noise of acquisition.

The events e(p, t) can then be defined by all the following information:

${e\left( {p,t} \right)} = \left\{ \begin{matrix} {p \in C \Subset {\mathbb{R}}^{2}} \\ {pol} \\ {I\left( {p,t} \right)} \end{matrix} \right.$

with C the spatial area of the sensor, pol the polarity representing the direction of the luminance change (for example, 1 for an increase or −1 for a decrease) and I(p, t) the light intensity signal of the point p at the moment t.

The light intensity signal can thus be all the pulse trains combined 104 such as described in FIG. 1. If t_(u) represents the time of the occurrence of the event, and the difference between t_(e+,u) and t_(e−,u) represents a value inversely proportional to the light intensity received, it is possible to code the intensity using three Diracs δ in such a way that I(p, t)=δ(t−t_(u))+δ(t−t_(e+,u))+δ(t−t_(e−,u)). The pixel intensity signal located at coordinate p thus enables luminance information to be coded temporally. This information can come directly from the electronic circuit of the sensor with a minimum amount of transformation.

FIG. 3 is an example of the calculation of a luminance component for two points p and q of two separate sensors u and v.

In order to determine if two points p and q of two sensors correspond to the same point of the scene observed, there is a hypothesis that the surfaces comprising the scene observed are Lambert surfaces (that is, surfaces of which the luminance is the same, whatever the angle of observation).

Consequently, for these surfaces, the intensity must be the same for the two sensors at one same moment, that is, I_(u)(p, t)=I_(v)(q, t).

It is, for example, possible to calculate a correlation between these two signals I_(u)(p, t) and I_(v)(q, t).

In order to be able to simply compare the Dirac-composed light intensity signals, it can be advantageous to convolute these signals by a non-void support core g_(σ)(t). It is then possible to calculate a correlation between the two signals I_(u)(p, t)*g_(σ)(t)=

(p, t) and I_(v)(q, t)*g_(σ)(t)=

(p, t).

Moreover, it can be useful to not limit the comparison of two single points, but to also consider the points located close to p and q (that is, located at a predetermined distance from p or q, distance in the mathematical sense of the term): all the points close to p define a set v_(u)(p) and all points close to q define a set v_(v)(q) (N is the cardinal of these sets). Thus, the luminance component can be expressed as follows:

$E_{I} = {1 - {\frac{1}{N}{\sum\limits_{\underset{q_{i} \in {v_{v}{(q)}}}{p_{i \in {v_{u}{(p)}}}}}\frac{\int_{- \infty}^{+ \infty}{\ \left( {p_{i},t} \right)\left( {q_{i},t} \right){dt}}}{\sqrt{\int_{- \infty}^{+ \infty}{\ \left( {p_{i},t} \right)^{2}{dt}}}\sqrt{\int_{- \infty}^{+ \infty}{\left( {q_{i},t} \right)^{2}{dt}}}}}}}$

Of course, it is possible to reduce the integration terminals by defining w the support of convoluted functions Ī(•) for all points located close to p or q such as defined above.

$E_{I} = {1 - {\frac{1}{N}{\sum\limits_{\underset{q_{i} \in {v_{v}{(q)}}}{p_{i} \in {v_{u}{(p)}}}}\; \frac{\int_{\omega}{\left( {p_{i},t} \right)\left( {q_{i},t} \right){dt}}}{\sqrt{\int_{\omega}{\left( {p_{i},t} \right)^{2}{dt}}}\sqrt{\int_{\omega}{\left( {q_{i},t} \right)^{2}{dt}}}}}}}$

Finally, it is possible to generalise this formula by using more than two sensors. For example, with Q sensors {u, v, w, . . . }, it is possible to write:

$E_{I} = {1 - {\frac{1}{\left( {Q - 1} \right)N}{\sum\limits_{\underset{\underset{c \in {\{{v,w,\ldots}\}}}{q_{ci} \in {v_{c}{(q_{c})}}}}{p_{i} \in {v_{u}{(p)}}}}\; \frac{\int_{\omega}{\left( {p_{i},t} \right)\left( {q_{ci},t} \right){dt}}}{\sqrt{\int_{\omega}{\left( {p_{i},t} \right)^{2}{dt}}}\sqrt{\int_{\omega}{\left( {q_{ci},t} \right)^{2}{dt}}}}}}}$

The core g_(σ)(t) is advantageously a Gaussian variance a. It can also be a door width function σ.

FIG. 4c μs a representation of cards 401 and 402 generated using separate asynchronous sensors.

To generate these cards, it is possible to define the function S as the sum, for each event

${{ev}_{p}(i)} = \left\{ \begin{matrix} {p_{i} = p} \\ {{pol}_{i} = {pol}} \\ t_{p} \end{matrix} \right.$

of a given pixel p and for a given polarity pol, at a given moment t, of the language primitive

${S_{prim}\left( {p,{pol},t} \right)} = \left\{ {\begin{matrix} {\max \left( {{h + {\theta \cdot \left( {t_{p} - t} \right)}},0} \right)} & {{{if}\mspace{14mu} t} \geq t_{p}} \\ 0 & {otherwise} \end{matrix},} \right.$

h being a predetermined value and θ being a predetermined factor corresponding to the speed of the decreasing of the language primitive.

The “sum” of the language primitive can also be seen mathematically as a convolution:

-   -   of the language primitive

${S_{prim}\left( {p,t} \right)} = \left\{ \begin{matrix} {\max \left( {{h - {\theta \cdot t}},0} \right)} & {{{if}\mspace{14mu} t} \geq 0} \\ 0 & {otherwise} \end{matrix} \right.$

(or more generally, any decreasing function),

-   -   with a signal comprising a Dirac for each time t, to which an         event ev_(p)(i)=

$\left\{ {\begin{matrix} {p_{i} = p} \\ {{pol}_{i} = {pol}} \\ t_{p} \end{matrix}\mspace{14mu} {{occurs}.}} \right.$

As an illustration, FIG. 4a shows three possible activity signals t→S, for three pixels p₁, p₂ and p₃ of the sensor (and for a given polarity value pol).

In the absence of events, the value of S(p₁, t), S(p₂, t) or S(p₃, t) is zero. However, at the time of the occurrence of a polarity event pol (for example, 410) at the level of the pixel p₁, S(p₁, t) takes a predetermined threshold value (here h, this value h could be unitary).

The value of the activity signal S(p₁, t) then progressively decreases after this event to reach towards 0.

This is the same for the event 411 for the pixel p₁, for the event 412 for the pixel p₂, or for the event 413/414 for the pixel p₃.

If the decrease of the activity signal S here is linear, it is possible to anticipate any type of decrease as an exponential decrease:

${h \cdot e} - \frac{t - t_{p}}{\tau}$

This exponential decrease can be illustrated by FIG. 4b (see the curve 4 b and the event 320).

Moreover, it is possible that, at the time of the occurrence of an event for the pixel considered (for example, p₄ here), the value of the function S is not negligible in relation to the value of h (for example, the event 421 is temporally close to the event 422).

In an embodiment, at the time of the occurrence of the later event 422, the value of the activity signal S can be set to the sum (possibly weighted) of the current value of S, just before the event 422 (that is, h₀) and of h. Thus, the decrease in the curve S will start from the value h+h₀ as FIG. 4b shows. Moreover, it is possible to anticipate that the value of h+h₀ is capped at a predetermined value h1 (that is, min(h₁, h+h₀)).

In another embodiment, at the time of the occurrence of the later event 422, the value of the curve S is set to the value h, whatever the value of h₀ (that is, the previous events to the last event (that is, the later event) are ignored). In this other embodiment, it is possible to define a time known as “last event time” defined as follows:

T(p,pol,i)=max(t _(j))|j<i

or

T(p,pol,t)=max(t _(j))|t _(j) <t

with t_(j) event times occurring for the pixel for a pixel p with the polarity pol.

Conceptually, p→T(p, pol, t) defines a time card of the last events of the same polarity that have occurred temporally, just before a reference time (that is, t).

It can therefore be defined, in this other embodiment, p→S(p, pol, t) as being as a function of this set of time T(p, pol, t).

For example, p→S(p, pol, t):

$\left. p\rightarrow{{S\left( {p,{pol},t} \right)}\text{:}\mspace{14mu} \left\{ \begin{matrix} {{h \cdot e} - \frac{t - {T\left( {p,{pol},t} \right)}}{\tau}} & {{{if}\mspace{14mu} t} \geq {T\left( {p,{pol},t} \right)}} \\ 0 & {otherwise} \end{matrix} \right.} \right.$

with τ and h a predetermined time constant (S can be any decreasing function with time t over an interval comprising as a lower terminal T(p, pol, t)).

The creation of a pixel card S representative of the “freshness” of events of these pixels is advantageous, as it enables a continuous and simple representation of discontinuous concepts (that is, events). This created card enables the representation of events to be transformed into a simple area of understanding.

Consequently, its creation simplifies the handling and comparison of events.

This function S is representative of a “freshness” of events that have occurred for this pixel.

The cards 401 and 402 of the FIG. 4c are representations of this function S for a given time t and for two asynchronous sensors capturing the movement of one same hand from two different viewpoints.

The darkest points represent the points of which the last events are the most recent in relation to time t (that is, having the largest S value).

The clearest points represent the points of which the last events are the most distant in relation to time t (that is, having the smallest S value, the background of the image is greyed to make the clear values stand out more easily, although the background corresponds to zero values of the function S).

The dispersed darkened points correspond to a sensor capture noise.

For each event occurring on the date t₀, it is possible to determine a movement card for the pixel p. Consequently, each pixel p of the card has, as a value, S(p, t₀).

In order to determine if two points p and q of two sensors correspond to the same point of the scene observed, it is assumed that the S value of the two sensors at the respective points p and q will be similar (this is not necessarily the case in certain limited situations), either S(p)=S(q) or at the least, S(p)≈S(q).

It is, for example, possible to calculate a correlation between these two values S(p) and S(q) Moreover, it can be useful to not limit the comparison of two single points, but to also consider the points located close to p (403) and q (404) (that is, located at a predetermined distance from p or q, a distance in the mathematical sense of the term): all the points close to p define a set v_(u)(p) (405) and all the points close to q define a set v_(v)(q) (406) (N is the cardinal of these sets).

It is possible to determine the correlation of two cards 405 and 406 close to points p and q. In addition, it is possible, in order to free a sensor from any time difference, to subtract their average respective optical flows 405 and 406 from each one (respectively S_(u) (p) and S_(v) (q)).

Thus, the movement component, for a given moment t, can be expressed as follows:

$E_{M} = {1 - {\frac{1}{2}{\sum\limits_{i = 1}^{N}\; \frac{\left( {{S_{u}\left( p_{i} \right)} - {\overset{\_}{S_{u}}(p)}} \right)\left( {{S_{v}\left( q_{i} \right)} - {\overset{\_}{S_{v}}(q)}} \right)}{{{{S_{u}\left( p_{i} \right)} - {\overset{\_}{S_{u}}(p)}}}{{{S_{v}\left( q_{i} \right)} - {\overset{\_}{S_{v}}(q)}}}}}}}$

with i the index of a point in the set v_(u)(p) and of a point in the set v_(v)(q).

Finally, it is possible to generalise this formula by using more than two sensors. For example, with Q sensors {u, v, w, . . . }, it is possible to write (by using the same notations as were used previously for the luminance component):

$E_{M} = {1 - {\frac{1}{Q - 1}{\sum\limits_{c \in {\{{v,w,\ldots}\}}}\; {\sum\limits_{i = 1}^{N}\; \frac{\left( {{S_{u}\left( p_{i} \right)} - {\overset{\_}{S_{u}}(p)}} \right)\left( {{S_{c}\left( q_{ci} \right)} - {\overset{\_}{S_{c}}\left( q_{c} \right)}} \right)}{{{{S_{u}\left( p_{i} \right)} - {\overset{\_}{S_{u}}(p)}}}{{{S_{c}\left( q_{ci} \right)} - {\overset{\_}{S_{c}}\left( q_{c} \right)}}}}}}}}$

FIGS. 5a and 5b are representations of examples of calculating geometric components in an embodiment of the invention.

When two sensors 501 and 502 are opposite one same scene (for example, a scene comprising the point X(t), see FIG. 5a ), and if the point p₁ ^(u) of the first sensor 501 representative of the point X(t) (that is, the point p₁ ^(u), X(t) and R_(u) are aligned), it is possible to define an epipolar straight line l_(uv) on the sensor 502.

R_(u) is the centre of projection of the sensor 501 and R_(v) is the centre of projection of the sensor 502.

This epipolar straight line l_(uv) is defined as being the intersection of the plane (X(t), R_(u), R_(v)) with the sensor 502.

More generally, a point p of the last sensor 501 defines an epipolar straight line l_(v)(p) on the second sensor 502 and a point q of the second sensor 502 defines an epipolar straight line l_(u)(q) on the first sensor 501.

Consequently, it is possible to define a geometric component for two points p and q of the first and second sensors:

$E_{G} = {\frac{1}{2\varepsilon_{g}}\left( {{d\left( {p,{l_{u}(q)}} \right)} + {d\left( {q,{l_{v}(p)}} \right)}} \right)}$

If the shooting device comprises three sensors (see FIG. 3b ), two epipolar straight lines can be defined per sensor, these straight lines being defined by the points considered on the two other sensors. Thus, these two epipolar straight lines then have an intersection, that below is called an epipolar intersection. Consequently:

-   -   a point p of the first sensor and a point q of the second sensor         define an epipolar intersection i_(w)(p, q) on the third sensor;     -   a point p of the first sensor and a point r of the third sensor         define an epipolar intersection i_(v)(p, r) on the second         sensor;     -   a point q of the second sensor and a point r of the third sensor         define an epipolar intersection i_(u)(q, r) on the first sensor.

Thus, it is possible to define a geometric component for three points p, q and r of the first, second and third sensors:

$E_{G} = {\frac{1}{3\varepsilon_{g}}\left( {{d\left( {p,{i_{u}\left( {q,r} \right)}} \right)} + {d\left( {q,{i_{v}\left( {p,r} \right)}} \right)} + {d\left( {r,{i_{w}\left( {p,q} \right)}} \right)}} \right)}$

with ε_(g) a predetermined value of a distance representative of a maximum acceptable geometric difference.

If the shooting device comprises more than three sensors (for example, Q sensors), it is possible to generalise the previous formula, by considering that the epipolar intersection of a sensor is the point located the closest to the set of epipolar straight lines defined on this sensor by current points of the other sensors (for example, by minimising the sum of the distances or by minimising the square of the distances of said points to the epipolar straight lines).

$E_{G} = {\frac{1}{Q_{\varepsilon_{g}}}\left( {\sum\limits_{c \in {\{{u,v,w,\ldots}\}}}{d\left( {p_{c},{i_{c}\left( \left\{ p_{d} \right\}_{d \in {{\{{u,v,w,\ldots}\}}\backslash c}} \right)}} \right)}} \right)}$

It is also possible to determine a time component for an event e(p, t_(u)) of the first sensor and an event e(q, t_(v)) of the second sensor:

$E_{T} = \frac{{t_{u} - t_{v}}}{\varepsilon_{t}}$

with ε_(t) a number that has the dimension of time and is representative of a maximum acceptable time difference between these two events.

If the shooting device comprises more than three sensors (for example, Q sensors), it is possible to generalise the previous formula:

$E_{T} = \frac{\sum\limits_{c = {\{{v,w,\ldots}\}}}\; {{t_{u} - t_{c}}}}{Q_{\varepsilon_{t}}}$

FIG. 6 illustrates a flowchart presenting an embodiment according to the invention.

Upon receipt of two sets of asynchronous events 601 and 602 coming from two separate asynchronous sensors, and compared with one same scene, it is possible to select two events from these sensors (step 603, defined by the pixel p_(i) and time t_(1i) for the first sensor and the pixel q_(j) and time t_(2j) for the second sensor).

Once these events are selected, it is possible to determine at least one component from amongst the following components, as stated above:

-   -   a geometric component (step 604);     -   a time component (step 605);     -   a movement component (step 606);     -   a luminance component (step 607).

For an event e₁(p_(i), t_(1i)) set for the first sensor, it is possible to iterate (test 608, output j+1) over a large number of events e₂(q_(j), t_(2j)) (by making the index j vary, for example): iterations can outline all the events of the second sensor or advantageously only a subset of these events (for example, only those located at a predetermined geometric distance from the epipolar straight line or from the epipolar intersection defined by at least p₁ and/or only those located at a predetermined time distance of time t_(1i)).

Once the iterations have ended (test 608, output OK), it is possible to determine the event e₂(q_(j), t_(2j)) minimising a cost function E for an event e₁(p_(i), t_(1i)) set (step 609). The cost function can be, for example, a simple sum (E=E_(T)+E_(M)+E_(G)+E_(I)) or a weighted sum (E=ω_(T)E_(T)+ω_(M)E_(M)+W_(G)E_(G)+ω_(I),E_(I)) of the components previously calculated (any other function involving these components is also possible).

It has been observed through experiments, that a cost function considering a luminance component and/or a movement component would enable the precision of 3D reconstructions made to be significantly increased.

Once the minimisation has been carried out, it is possible to connect the points p_(i) and q_(j) (step 610), and thus calculate the distances or the position in the space of the point X(t) of the scene observed, representative of the connected points p_(i) and q_(j) (step 611).

The distances calculated (or the position of the point X(t) in the space) are then returned (612).

FIG. 7 illustrates a device for implementing an embodiment according to the invention.

In this embodiment, the device comprises a computer 700, comprising a memory 705 to store instructions enabling the implementation of the method, data from measurements received, and time data to carry out different steps of the method such as described previously.

The computer additionally comprises a circuit 704. This circuit can be, for example:

-   -   a processor able to interpret instructions in the form of a         computer program, or     -   an electronic card of which the steps of the method of the         invention are described in the silicon chip, or again     -   a programmable electronic chip, like a FPGA (“Field-Programmable         Gate Array”) chip.

This computer comprises an input interface 703 to receive events from sensors, and an output interface 706 for the supply of distances 707. Finally, the computer can comprise, to enable easy interaction with a user, a screen 701 and a keyboard 702. Of course, the keyboard is optional, in particular as part of a computer having the form of a touchscreen tablet, for example.

Moreover, the functional diagram presented in FIG. 6 is a typical example of a program of which certain instructions can be made for the device described. In this regard, FIG. 6 can correspond to the flowchart of the general algorithm of a computer program in the sense of the invention.

Of course, the present invention is not limited to the forms of embodiment described above as an example; it extends to other variants.

Other embodiments are possible.

For example, the flowchart of FIG. 6 can also comprise an iteration on the events e₁(p_(i), t_(1i)) in order to connect several events of the first sensor with events of the second sensor. 

1. Method of 3D reconstruction of a scene, the method comprising: reception (601) of a first piece of asynchronous information from a first sensor (501) that has a first pixel matrix positioned opposite the scene, the first piece of asynchronous information comprising, for each pixel (p) of the first matrix, the first successive events coming from said pixel; reception (602) of a second piece of asynchronous information from a second sensor (502) that has a second pixel matrix positioned opposite the scene, the second piece of asynchronous information comprising, for each pixel (q) of the second matrix, the second successive events coming from said pixel, the second sensor being separate from the first sensor; matching (610) of a first event from amongst the first successive events with a second event from amongst the second successive events depending on a minimisation (609) of a cost function (E); wherein the cost function comprises at least one component from amongst: a luminance component (E_(T)), said luminance component depending on at least: a first luminance signal (I_(u)) coming from a pixel of the first sensor convoluted with a convolution core (g_(σ)(t)), the luminance of said pixel depending on a difference between the maximums (t_(e−,u),t_(e+,u)) of said first signal; and a second luminance signal (I_(v)) coming from a pixel of the second sensor convoluted with said convolution core, the luminance of said pixel depending on a difference between the maximums (t_(e−,v),t_(e+,v)) of said second signal; a movement component (E_(M)), said movement component depending on at least: time values relating to the occurrence of events spatially located at a predetermined distance from a pixel of the first sensor; time values relating to the occurrence of events spatially located at a predetermined distance from a pixel of the second sensor.
 2. Method according to claim 1, wherein the cost function (E) additionally comprises: a time component (E_(T)), said time component depending on a difference between: a time value relating to an event of the first sensor; a time value relating to an event of the second sensor.
 3. Method according to claim 1, wherein the cost function (E) additionally comprises: a geometric component (E_(G)), said geometric component depending on: a spatial distance from a pixel of the second sensor at an epipolar straight line or at an epipolar intersection defined by at least one pixel of the first sensor.
 4. Method according to claim 1, wherein, the luminance signal (I_(u), I_(v)) of the pixel of the first sensor and of the pixel of the second sensor comprising a maximum, coding an occurrence time of a luminance variation, the convolution core is a predetermined Gaussian variance.
 5. Method according to claim 1, wherein said luminance component (E_(I)) additionally depends on: luminance signals of pixels of the first sensor, spatially located at a predetermined distance from the first pixel of the first sensor, convoluted with the convolution core; and luminance signals of pixels of the second sensor, spatially located at a predetermined distance from the second pixel of the second sensor, convoluted with the convolution core.
 6. Method according to claim 1, wherein said movement component (E_(M)) depends on: an average value (S(p)) of the time values relating to the occurrence of pixel events of the first sensor, spatially located at a predetermined distance from the pixel of the first sensor; an average value (S(q)) of the time values relating to the occurrence of pixel events of the second sensor, spatially located at a predetermined distance from the pixel of the second sensor.
 7. Method according to claim 1, wherein said movement component (E_(M)) depends on, for a given time: for each current time value relating to the occurrence of events, spatially located at a predetermined distance from a pixel of the first sensor, of a function value decreasing from a distance of said given time to said current time value; for each current time value relating to the occurrence of events, spatially located at a predetermined distance from a pixel of the second sensor, of a function value decreasing from a distance of said given time to said current time value.
 8. Method according to claim 1, wherein said movement component (E_(M)) depends on: a first convolution of a decreasing function with a signal comprising a Dirac for each time value relating to the occurrence of events, spatially located at a predetermined distance from a pixel of the first sensor; a second convolution of a decreasing function with a signal comprising a Dirac for each time value relating to the occurrence of events, spatially located at a predetermined distance from a pixel of the second sensor.
 9. Device for the 3D reconstruction of a scene, the method comprising: an interface (703) for the reception (601) of a first piece of asynchronous information from a first sensor (501) that has a first pixel matrix positioned opposite the scene, the first piece of asynchronous information comprising, for each pixel (p) of the first matrix, the first successive events coming from said pixel; an interface (703) for the reception (602) of a second piece of asynchronous information from a second sensor (502) that has a second pixel matrix positioned opposite the scene, the second piece of asynchronous information comprising, for each pixel (q) of the second matrix, the second successive events coming from said pixel, the second sensor being separate from the first sensor; a processor (704) suitable for the matching (610) of a first event from amongst the first successive events with a second event from amongst the second successive events depending on a minimisation (609) of a cost function (E); wherein the cost function comprises at least one component from amongst: a luminance component (E_(I)), said luminance component depending on at least: a first luminance signal (I_(u)) coming from a pixel of the first sensor, convoluted with a convolution core (g_(σ)(t)), the luminance of said pixel depending on a difference between the maximums (t_(e−,u),t_(e+,u)) of said first signal; and a second luminance signal (I_(v)) coming from a pixel of the second sensor, convoluted with said convolution core, the luminance of said pixel depending on a difference between the maximums (t_(e−,v),t_(e+,v)) of said second signal; a movement component (E_(M)), said movement component depending on at least: time values relating to the occurrence of events, spatially located at a predetermined distance from a pixel of the first sensor; time values relating to the occurrence of event, spatially located at a predetermined distance from a pixel of the second sensor.
 10. Computer program product comprising instructions for the implementation of the method according to claim 1, when this program is executed by a processor.
 11. Method according to claim 2, wherein the cost function (E) additionally comprises: a geometric component (E_(G)), said geometric component depending on: a spatial distance from a pixel of the second sensor at an epipolar straight line or at an epipolar intersection defined by at least one pixel of the first sensor.
 12. Method according to claim 2, wherein, the luminance signal (I_(u), I_(v)) of the pixel of the first sensor and of the pixel of the second sensor comprising a maximum, coding an occurrence time of a luminance variation, the convolution core is a predetermined Gaussian variance.
 13. Method according to claim 3, wherein, the luminance signal (I_(u), I_(v)) of the pixel of the first sensor and of the pixel of the second sensor comprising a maximum, coding an occurrence time of a luminance variation, the convolution core is a predetermined Gaussian variance.
 14. Method according to claim 2, wherein said luminance component (E_(I)) additionally depends on: luminance signals of pixels of the first sensor, spatially located at a predetermined distance from the first pixel of the first sensor, convoluted with the convolution core; and luminance signals of pixels of the second sensor, spatially located at a predetermined distance from the second pixel of the second sensor, convoluted with the convolution core.
 15. Method according to claim 3, wherein said luminance component (E_(I)) additionally depends on: luminance signals of pixels of the first sensor, spatially located at a predetermined distance from the first pixel of the first sensor, convoluted with the convolution core; and luminance signals of pixels of the second sensor, spatially located at a predetermined distance from the second pixel of the second sensor, convoluted with the convolution core.
 16. Method according to claim 4, wherein said luminance component (E_(I)) additionally depends on: luminance signals of pixels of the first sensor, spatially located at a predetermined distance from the first pixel of the first sensor, convoluted with the convolution core; and luminance signals of pixels of the second sensor, spatially located at a predetermined distance from the second pixel of the second sensor, convoluted with the convolution core.
 17. Method according to claim 2, wherein said movement component (E_(M)) depends on: an average value (S(p)) of the time values relating to the occurrence of pixel events of the first sensor, spatially located at a predetermined distance from the pixel of the first sensor; an average value (S(q)) of the time values relating to the occurrence of pixel events of the second sensor, spatially located at a predetermined distance from the pixel of the second sensor.
 18. Method according to claim 3, wherein said movement component (E_(M)) depends on: an average value (S(p)) of the time values relating to the occurrence of pixel events of the first sensor, spatially located at a predetermined distance from the pixel of the first sensor; an average value (S(q)) of the time values relating to the occurrence of pixel events of the second sensor, spatially located at a predetermined distance from the pixel of the second sensor.
 19. Method according to claim 4, wherein said movement component (E_(M)) depends on: an average value (S(p)) of the time values relating to the occurrence of pixel events of the first sensor, spatially located at a predetermined distance from the pixel of the first sensor; an average value (S(q)) of the time values relating to the occurrence of pixel events of the second sensor, spatially located at a predetermined distance from the pixel of the second sensor.
 20. Method according to claim 5, wherein said movement component (E_(M)) depends on: an average value (S(p)) of the time values relating to the occurrence of pixel events of the first sensor, spatially located at a predetermined distance from the pixel of the first sensor; an average value (S(q)) of the time values relating to the occurrence of pixel events of the second sensor, spatially located at a predetermined distance from the pixel of the second sensor. 